{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "99c810c345144611a574b0354978952d": {
          "model_module": "@jupyter-widgets/output",
          "model_name": "OutputModel",
          "model_module_version": "1.0.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/output",
            "_model_module_version": "1.0.0",
            "_model_name": "OutputModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/output",
            "_view_module_version": "1.0.0",
            "_view_name": "OutputView",
            "layout": "IPY_MODEL_9beb2254b21c408c905f63b635acef79",
            "msg_id": "",
            "outputs": [
              {
                "output_type": "display_data",
                "data": {
                  "text/plain": "\u001b[38;2;106;0;255m⠇\u001b[0m ✨ You're running DeepEval's latest \u001b[38;2;106;0;255mFaithfulness Metric\u001b[0m! \u001b[38;2;55;65;81m(using gpt-4o, strict=False, async_mode=True)...\u001b[0m\n",
                  "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #6a00ff; text-decoration-color: #6a00ff\">⠇</span> ✨ You're running DeepEval's latest <span style=\"color: #6a00ff; text-decoration-color: #6a00ff\">Faithfulness Metric</span>! <span style=\"color: #374151; text-decoration-color: #374151\">(using gpt-4o, strict=False, async_mode=True)...</span>\n</pre>\n"
                },
                "metadata": {}
              }
            ]
          }
        },
        "9beb2254b21c408c905f63b635acef79": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e1228e602142478db648aad0ff915991": {
          "model_module": "@jupyter-widgets/output",
          "model_name": "OutputModel",
          "model_module_version": "1.0.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/output",
            "_model_module_version": "1.0.0",
            "_model_name": "OutputModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/output",
            "_view_module_version": "1.0.0",
            "_view_name": "OutputView",
            "layout": "IPY_MODEL_795101f3f9d242fd9633bbbc8f36f08e",
            "msg_id": "",
            "outputs": [
              {
                "output_type": "display_data",
                "data": {
                  "text/plain": "\u001b[38;2;106;0;255m⠙\u001b[0m ✨ You're running DeepEval's latest \u001b[38;2;106;0;255mFaithfulness Metric\u001b[0m! \u001b[38;2;55;65;81m(using gpt-4o, strict=False, async_mode=True)...\u001b[0m\n",
                  "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #6a00ff; text-decoration-color: #6a00ff\">⠙</span> ✨ You're running DeepEval's latest <span style=\"color: #6a00ff; text-decoration-color: #6a00ff\">Faithfulness Metric</span>! <span style=\"color: #374151; text-decoration-color: #374151\">(using gpt-4o, strict=False, async_mode=True)...</span>\n</pre>\n"
                },
                "metadata": {}
              }
            ]
          }
        },
        "795101f3f9d242fd9633bbbc8f36f08e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8afc66dbf89641aba9f4041b8b334eee": {
          "model_module": "@jupyter-widgets/output",
          "model_name": "OutputModel",
          "model_module_version": "1.0.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/output",
            "_model_module_version": "1.0.0",
            "_model_name": "OutputModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/output",
            "_view_module_version": "1.0.0",
            "_view_name": "OutputView",
            "layout": "IPY_MODEL_90b6385e7acb42878af9995a46789a2f",
            "msg_id": "",
            "outputs": [
              {
                "output_type": "display_data",
                "data": {
                  "text/plain": "\u001b[38;2;106;0;255m⠏\u001b[0m ✨ You're running DeepEval's latest \u001b[38;2;106;0;255mContextual Relevancy Metric\u001b[0m! \u001b[38;2;55;65;81m(using gpt-4o, strict=False, async_mode=True)...\u001b[0m\n",
                  "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #6a00ff; text-decoration-color: #6a00ff\">⠏</span> ✨ You're running DeepEval's latest <span style=\"color: #6a00ff; text-decoration-color: #6a00ff\">Contextual Relevancy Metric</span>! <span style=\"color: #374151; text-decoration-color: #374151\">(using gpt-4o, strict=False, async_mode=True)...</span>\n</pre>\n"
                },
                "metadata": {}
              }
            ]
          }
        },
        "90b6385e7acb42878af9995a46789a2f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7cbc0bc6cd564bc1b8153b6ec1742cd4": {
          "model_module": "@jupyter-widgets/output",
          "model_name": "OutputModel",
          "model_module_version": "1.0.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/output",
            "_model_module_version": "1.0.0",
            "_model_name": "OutputModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/output",
            "_view_module_version": "1.0.0",
            "_view_name": "OutputView",
            "layout": "IPY_MODEL_017397fb1d264a55a5aff42cd2600226",
            "msg_id": "",
            "outputs": [
              {
                "output_type": "display_data",
                "data": {
                  "text/plain": "\u001b[38;2;106;0;255m⠼\u001b[0m ✨ You're running DeepEval's latest \u001b[38;2;106;0;255mContextual Relevancy Metric\u001b[0m! \u001b[38;2;55;65;81m(using gpt-4o, strict=False, async_mode=True)...\u001b[0m\n",
                  "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #6a00ff; text-decoration-color: #6a00ff\">⠼</span> ✨ You're running DeepEval's latest <span style=\"color: #6a00ff; text-decoration-color: #6a00ff\">Contextual Relevancy Metric</span>! <span style=\"color: #374151; text-decoration-color: #374151\">(using gpt-4o, strict=False, async_mode=True)...</span>\n</pre>\n"
                },
                "metadata": {}
              }
            ]
          }
        },
        "017397fb1d264a55a5aff42cd2600226": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9ae13990f4b94589bedfecf87ccfae38": {
          "model_module": "@jupyter-widgets/output",
          "model_name": "OutputModel",
          "model_module_version": "1.0.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/output",
            "_model_module_version": "1.0.0",
            "_model_name": "OutputModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/output",
            "_view_module_version": "1.0.0",
            "_view_name": "OutputView",
            "layout": "IPY_MODEL_569d02b2e1b14d639d0341fb1cb96e8f",
            "msg_id": "",
            "outputs": [
              {
                "output_type": "display_data",
                "data": {
                  "text/plain": "\u001b[38;2;106;0;255m⠏\u001b[0m ✨ You're running DeepEval's latest \u001b[38;2;106;0;255mAnswer Relevancy Metric\u001b[0m! \u001b[38;2;55;65;81m(using gpt-4o, strict=False, async_mode=True)...\u001b[0m\n",
                  "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #6a00ff; text-decoration-color: #6a00ff\">⠏</span> ✨ You're running DeepEval's latest <span style=\"color: #6a00ff; text-decoration-color: #6a00ff\">Answer Relevancy Metric</span>! <span style=\"color: #374151; text-decoration-color: #374151\">(using gpt-4o, strict=False, async_mode=True)...</span>\n</pre>\n"
                },
                "metadata": {}
              }
            ]
          }
        },
        "569d02b2e1b14d639d0341fb1cb96e8f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b7e41a0b97244238ad271140f9b7755e": {
          "model_module": "@jupyter-widgets/output",
          "model_name": "OutputModel",
          "model_module_version": "1.0.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/output",
            "_model_module_version": "1.0.0",
            "_model_name": "OutputModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/output",
            "_view_module_version": "1.0.0",
            "_view_name": "OutputView",
            "layout": "IPY_MODEL_029cbff8d3ab423aaf51d2a62f285bf1",
            "msg_id": "",
            "outputs": [
              {
                "output_type": "display_data",
                "data": {
                  "text/plain": "\u001b[38;2;106;0;255m⠋\u001b[0m ✨ You're running DeepEval's latest \u001b[38;2;106;0;255mAnswer Relevancy Metric\u001b[0m! \u001b[38;2;55;65;81m(using gpt-4o, strict=False, async_mode=True)...\u001b[0m\n",
                  "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #6a00ff; text-decoration-color: #6a00ff\">⠋</span> ✨ You're running DeepEval's latest <span style=\"color: #6a00ff; text-decoration-color: #6a00ff\">Answer Relevancy Metric</span>! <span style=\"color: #374151; text-decoration-color: #374151\">(using gpt-4o, strict=False, async_mode=True)...</span>\n</pre>\n"
                },
                "metadata": {}
              }
            ]
          }
        },
        "029cbff8d3ab423aaf51d2a62f285bf1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        }
      }
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "**How to use Query transformation and Hypothetical answering to re-rank retrieved articles and enhance the performance of your RAG pipeline?**"
      ],
      "metadata": {
        "id": "2YVPjLN9cVAU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "[Hanane DUPOUY LinkedIn](https://https://www.linkedin.com/in/hanane-d-algo-trader/): https://www.linkedin.com/in/hanane-d-algo-trader/"
      ],
      "metadata": {
        "id": "IjEguVl88FjA"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Ever wondered how to accurately answer the question, `'What impact did the global outage of CrowdStrike, extensively used by Microsoft, have on Microsoft's stock price?'`\n",
        "\n",
        "To tackle this, we'll employ the **hypothetical answer re-ranking** technique:\n",
        "\n",
        "\n",
        "-**GOAL**: I'll evaluate whether the **hypothetical answer** can improve the re-ranking and the retrieved context, and subsequently enhance the LLM's response within our RAG pipeline. Alternatively, we will assess if the original query alone is sufficient to retrieve the appropriate context and deliver accurate results.\n",
        "\n",
        "- Different techniques will be employed: **Query Transformation, Hypothetical Answers, Embeddings, and Similarity Scoring** to retrieve the relevant context from news articles fetched from the NEWS API.\n",
        "\n",
        "\n",
        "- In these techniques, we will compare the **capabilities of three LLMs**: **gpt-4o-mini** (the latest small model from OpenAI), **gpt-4o** (the most capable LLM from OpenAI), and **gpt-3.5-turbo**.\"\n",
        "\n",
        "\n",
        "- I'll use use **3 evaluation metrics** from **deepEval** for RAG pipelines: **Faithfulness, Context Relevancy and Answer relevancy**.\n",
        "These metrics are explained in the notebook.\n",
        "\n",
        "- We will compare the three LLMs using two techniques: **Hypothetical Answer Re-ranking vs. Original Query Retrieval**."
      ],
      "metadata": {
        "id": "ziSY8wNYkHLQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Steps:**\n",
        "\n",
        "For each of the three LLMs: **GPT-4o-mini, GPT-4o, and GPT-3.5-turbo**:\n",
        "\n",
        "**1-** We will perform **Search Queries** (or **Query transformation**) using the LLM to generate various formulation with the same keywords from the original user query.\n",
        "\n",
        "**2-** We will use an LLM to generate a **hypothetical answer**. This creative response will serve as a potential answer, using placeholders instead of actual facts.\n",
        "\n",
        "**3- **Based on each query from the search queries (1-), we will retrieve news article from NEWS API.\n",
        "\n",
        "**4-** We will **embedd** user query, hypothetical answer and the collected articles\n",
        "\n",
        "**5-** We compute the **similarity score** between 2 sets:\n",
        "\n",
        "  5-1- Hypothetical answer (2-) vs retrieved context (3-)\n",
        "\n",
        "  5-1- Original query (2-) vs retrieved context (3-)\n",
        "\n",
        "**6- **Ask the LLM to give the final answer based on the user query and the retrieved context\n",
        "\n",
        "**7-** Use 3 evalutaions metrics from DeepVal to evaluate the RAG pipeline: **Faithfulness, Context Relevancy and Answer relevancy.**\n",
        "\n",
        "**8-** Key Takeways\n"
      ],
      "metadata": {
        "id": "thVYKZfbfkFy"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Install Lib"
      ],
      "metadata": {
        "id": "JSDlVKdkwRfZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import json"
      ],
      "metadata": {
        "id": "sR9R3NACmxls"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2Xl1_XIbmm1D"
      },
      "outputs": [],
      "source": [
        "!pip install openai\n",
        "from google.colab import userdata\n",
        "openai_api_key = userdata.get('OPENAI_API_KEY')\n",
        "news_api_key = userdata.get('NEWS_API_KEY')\n",
        "\n",
        "from openai import OpenAI\n",
        "client = OpenAI(api_key=openai_api_key)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Chat method OpenAI"
      ],
      "metadata": {
        "id": "ynq6pzcuwT64"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def get_completion_gpt(input, gpt_model = \"gpt-3.5-turbo\"):\n",
        "  completion = client.chat.completions.create(\n",
        "        model=gpt_model,\n",
        "        messages=[\n",
        "            {\"role\": \"system\", \"content\": \"Output only valid JSON\"},\n",
        "            {\"role\": \"user\", \"content\": input},\n",
        "        ],\n",
        "        response_format={ \"type\": \"json_object\" }\n",
        "    )\n",
        "\n",
        "  text = completion.choices[0].message.content\n",
        "  parsed = json.loads(text)\n",
        "  return parsed"
      ],
      "metadata": {
        "id": "GDXktg4Bmt9U"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Search NEWS API"
      ],
      "metadata": {
        "id": "paqVeXGKy6H3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "\n",
        "def search_news(query, news_api_key= news_api_key,num_articles=5, from_datetime = \"2024-07-18\",to_datetime = \"2024-07-21\"):\n",
        "    response = requests.get(\n",
        "        \"https://newsapi.org/v2/everything\",\n",
        "        params={\n",
        "            \"q\": query,\n",
        "            \"apiKey\": news_api_key,\n",
        "            \"pageSize\": num_articles,\n",
        "            \"sortBy\": \"relevancy\",\n",
        "            \"from\": from_datetime,\n",
        "            \"to\": to_datetime,\n",
        "        },\n",
        "    )\n",
        "\n",
        "    return response.json()"
      ],
      "metadata": {
        "id": "hRS4dKWYy4Ww"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Generate Search Queries"
      ],
      "metadata": {
        "id": "60L1oojrwbeI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "user_query = \"What impact did the global outage of CrowdStrike, which is used extensively by Microsoft, have on Microsoft's stock price?\"\n",
        "\n",
        "input = f\"\"\"\n",
        "You have access to a NEWS API that returns recent news articles related to the user's question.\n",
        "\n",
        "1. Make a list of search queries that match the topic described in the user's question.\n",
        "2. Use different keywords related to the topic to create a variety of queries, making some general and others more specific.\n",
        "3. Be imaginative and generate as many queries as possible. More queries will help you find better results.\n",
        "4. Pick 10 of these queries.\n",
        "For example, you can include queries like ['keyword_1 keyword_2', 'keyword_1', 'keyword_2'].\n",
        "\n",
        "# User question: {user_query}\n",
        "\n",
        "# Format: {{\"queries\": [\"query_1\", \"query_2\", \"query_3\"]}}\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "llms = [\"gpt-3.5-turbo\", \"gpt-4o-mini\", \"gpt-4o\"]\n",
        "\n",
        "dict_questions = {}\n",
        "for llm in llms:\n",
        "  print(llm)\n",
        "  parsed = get_completion_gpt(input, gpt_model = llm)\n",
        "  dict_questions[llm] = parsed"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "h6K5h05Dm8cL",
        "outputId": "69c1e753-4bc2-4964-f590-f8be0d66f015"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "gpt-3.5-turbo\n",
            "gpt-4o-mini\n",
            "gpt-4o\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "dict_questions"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PqwOXh1SqQbI",
        "outputId": "e6b924c9-ad5b-494d-d617-752a39abd491"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'gpt-3.5-turbo': {'queries': ['CrowdStrike global outage impact on Microsoft stock price',\n",
              "   'Microsoft stock price reaction to CrowdStrike global issue',\n",
              "   'CrowdStrike downtime effects on Microsoft shares',\n",
              "   'Microsoft stock performance after CrowdStrike service disruption',\n",
              "   'CrowdStrike outage influence on Microsoft stock value',\n",
              "   'Microsoft stock price correlation with CrowdStrike downtime',\n",
              "   'CrowdStrike incident impact on Microsoft share value',\n",
              "   'Microsoft stock response to CrowdStrike global service failure',\n",
              "   'CrowdStrike outage consequences on Microsoft stock market',\n",
              "   'Microsoft stock price fluctuation due to CrowdStrike global problem']},\n",
              " 'gpt-4o-mini': {'queries': ['CrowdStrike outage Microsoft stock impact',\n",
              "   'global outage CrowdStrike Microsoft',\n",
              "   'Microsoft stock price reaction CrowdStrike incident',\n",
              "   'CrowdStrike service disruption effects on Microsoft',\n",
              "   'impact of cybersecurity outages on stock prices',\n",
              "   'Microsoft financial performance CrowdStrike downtime',\n",
              "   'CrowdStrike incident analysis Microsoft equities',\n",
              "   'how CrowdStrike affects Microsoft stock valuation',\n",
              "   'effects of CrowdStrike on investor confidence in Microsoft',\n",
              "   'cybersecurity outages and stock market trends']},\n",
              " 'gpt-4o': {'queries': ['CrowdStrike global outage impact on Microsoft',\n",
              "   'Microsoft stock price after CrowdStrike outage',\n",
              "   'CrowdStrike outage effect on Microsoft',\n",
              "   'CrowdStrike downtime Microsoft stock',\n",
              "   'Microsoft shares after CrowdStrike issues',\n",
              "   'Global outage of CrowdStrike affecting Microsoft',\n",
              "   'CrowdStrike Microsoft stock market reaction',\n",
              "   'Impact of CrowdStrike outage on MSFT stock',\n",
              "   'How CrowdStrike outage influenced Microsoft',\n",
              "   'CrowdStrike problems Microsoft financial impact']}}"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Create a Hypothetical answer"
      ],
      "metadata": {
        "id": "PDbVHrB8xUTR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "hypoth_answer = f\"\"\"\n",
        "Make up an answer to the user's question. We'll use this fabricated answer to sort the search results.\n",
        "Imagine you have all the details to answer, but don't use real facts. Do not give any numbers.\n",
        "Instead, use placeholders like 'EVENT affected something,' 'NAME mentioned something on DATE,' or 'EVENT has caused something.'\n",
        "\n",
        "User question: {user_query}\n",
        "\n",
        "Format: {{\"hypotheticalAnswer\": \"hypothetical answer text\"}}\n",
        "\"\"\"\n",
        "\n",
        "print(hypoth_answer)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ByTLCPEYxW9o",
        "outputId": "fa08ce77-add9-43c7-9994-7cb4b781c401"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Make up an answer to the user's question. We'll use this fabricated answer to sort the search results. \n",
            "Imagine you have all the details to answer, but don't use real facts. Do not give any numbers.\n",
            "Instead, use placeholders like 'EVENT affected something,' 'NAME mentioned something on DATE,' or 'EVENT has caused something.'\n",
            "\n",
            "User question: What impact did the global outage of CrowdStrike, which is used extensively by Microsoft, have on Microsoft's stock price?\n",
            "\n",
            "Format: {\"hypotheticalAnswer\": \"hypothetical answer text\"}\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Trying differenet llms:\n",
        "\n",
        "hypoth_answer_llms = {}\n",
        "for llm in llms:\n",
        "  # print(llm)\n",
        "  parsed_hypothet_answer = get_completion_gpt(hypoth_answer, gpt_model = llm)\n",
        "  hypoth_answer_llms[llm] = parsed_hypothet_answer['hypotheticalAnswer']\n",
        "  print(f\"{llm}\\n {hypoth_answer_llms[llm]}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DI5PoRbXxq84",
        "outputId": "2969da82-f1a0-40c8-fefa-ed02c1bf8289"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "gpt-3.5-turbo\n",
            "gpt-3.5-turbo\n",
            " The global outage of CrowdStrike has caused a temporary dip in Microsoft's stock price as investors reacted to the uncertainty surrounding the cybersecurity risks. However, Microsoft's resilient business model helped to recover the losses in the following days.\n",
            "gpt-4o-mini\n",
            "gpt-4o-mini\n",
            " The global outage of CrowdStrike, which is extensively used by Microsoft, led to increased investor concerns over cybersecurity vulnerabilities, causing a temporary decline in Microsoft's stock price. Analysts noted that the EVENT raised questions about the reliability of third-party security services, and NAME mentioned something about potential risks on DATE. This situation has caused a ripple effect in the tech market, affecting investor confidence in companies reliant on CrowdStrike.\n",
            "gpt-4o\n",
            "gpt-4o\n",
            " The global outage of CrowdStrike, which is used extensively by Microsoft, caused immediate concerns among investors regarding the cybersecurity resilience of Microsoft's services. This EVENT brought about uncertainty in the market, leading to a brief downturn in Microsoft’s stock price. Analysts cited fears over potential vulnerabilities and service disruptions as primary reasons for the stock's volatility during this period. However, NAME mentioned on DATE that the company is taking necessary steps to mitigate any long-term impacts, which has since helped stabilize the stock price.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Fetch news articles from NEWS API for each query:"
      ],
      "metadata": {
        "id": "BkzeKQEnqohC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def get_articles_from_news_api(queries):\n",
        "  articles = []\n",
        "  for query in queries:\n",
        "    result = search_news(query)\n",
        "    if result['status'] == 'ok':\n",
        "      articles = articles + result['articles']\n",
        "    else:\n",
        "      raise Exception(result[\"message\"])\n",
        "  return articles"
      ],
      "metadata": {
        "id": "NsusZSF9zX3p"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "articles={}\n",
        "for llm in llms:\n",
        "  queries = dict_questions[llm]['queries']\n",
        "  queries.append(user_query)\n",
        "  articles[llm] =  get_articles_from_news_api(queries)\n",
        "  if articles[llm]!=None:\n",
        "    articles[llm] = list({article[\"url\"]: article for article in articles[llm]}.values())"
      ],
      "metadata": {
        "id": "Y1puWVqszD4Q"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for llm in llms:\n",
        "  print(len(articles[llm]))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "S0Dw8l5d0Pv3",
        "outputId": "e1e55e9e-e833-4e0c-c075-c0aae5f8c01b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "26\n",
            "23\n",
            "40\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# #To save data locally\n",
        "# for llm in llms:\n",
        "#   pd.DataFrame(articles[llm]).to_csv(\"articles_\"+llm+\".csv\")"
      ],
      "metadata": {
        "id": "faIMg0BtUbn2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Display some articles:\n",
        "print(\"Total number of articles:\", len(articles)) #3 LLM ==> 3 set of articles\n",
        "llm = llms[-1]\n",
        "for article in articles[llm][0:5]:\n",
        "    print(\"Title:\", article[\"title\"])\n",
        "    print(\"Url:\", article[\"url\"])\n",
        "    print(\"Description:\", article[\"description\"])\n",
        "    print(\"Content:\", article[\"content\"][0:300] + \"...\")\n",
        "    print()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "P3uy09GTrHkq",
        "outputId": "9a163b92-96af-4c91-8c03-d4b2d8a305bb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total number of articles: 3\n",
            "Title: The Global CrowdStrike Outage Triggered a Surprise Return to Cash\n",
            "Url: https://www.wired.com/story/microsoft-crowdstrike-outage-cash/\n",
            "Description: The event caused chaos at airports, grocery stores, and Starbucks outlets.\n",
            "Content: On Friday, when a CrowdStrike update caused millions of Microsoft systems to crash around the world, many businesses were faced with a choice: Go cash-only, or close until systems came back online.\r\n",
            "… [+2941 chars]...\n",
            "\n",
            "Title: Huge Microsoft Outage, Linked to CrowdStrike, Takes Down Computers Around the World\n",
            "Url: https://www.wired.com/story/microsoft-windows-outage-crowdstrike-global-it-probems/\n",
            "Description: A software update from cybersecurity company Crowdstrike appears to have inadvertently disrupted IT systems globally.\n",
            "Content: Banks, airports, TV stations, hotels, and countless other businesses are all facing widespread IT outages, leaving flights grounded and causing widespread disruption, after Windows machines have disp… [+1941 chars]...\n",
            "\n",
            "Title: Chaos Reigns as Global Outage Breaks Everything from Airlines to Emergency Services\n",
            "Url: https://gizmodo.com/chaos-reigns-as-global-outage-breaks-everything-from-airlines-to-emergency-services-2000476734\n",
            "Description: Microsoft confirmed there’s an ongoing outage connected to IT company CloudStrike. There's a patch, though we're still feeling the impacts.\n",
            "Content: The whole world was thrown for a loop Friday after Microsoft confirmed theres a humongous, ongoing outage connected to the IT security company CrowdStrike. While the makers of Windows said the underl… [+2929 chars]...\n",
            "\n",
            "Title: How One Bad CrowdStrike Update Crashed the World’s Computers\n",
            "Url: https://www.wired.com/story/crowdstrike-outage-update-windows/\n",
            "Description: A defective CrowdStrike kernel driver sent computers around the globe into a reboot death spiral, taking down air travel, hospitals, banks, and more with it. Here’s how that’s possible.\n",
            "Content: That deeper access also introduces a far higher possibility that security softwareand updates to that softwarewill crash the whole system, says Matthieu Suiche, head of detection engineering at the s… [+3010 chars]...\n",
            "\n",
            "Title: Global services slowly recovering after bug causes IT chaos\n",
            "Url: https://www.bbc.com/news/articles/cg3m4jgdprxo\n",
            "Description: The incident has sparked concern over the vulnerability of the world's interconnected technologies.\n",
            "Content: By Robert Greenall, BBC News\r\n",
            "The outage has caused major delays at airports around the world\r\n",
            "Businesses and services around the world are slowly recovering after a massive IT outage affected comput… [+3598 chars]...\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Embeddings and cosine similarity"
      ],
      "metadata": {
        "id": "RJ-R5HO80ptu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Methods"
      ],
      "metadata": {
        "id": "9k3iEBkn0wH-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def get_embeddings(input):\n",
        "    response = client.embeddings.create(model=\"text-embedding-ada-002\", input=input)\n",
        "    return [data.embedding for data in response.data]"
      ],
      "metadata": {
        "id": "S1_t7_v9sdgL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def get_embeddings_articles(articles):\n",
        "  articles_prepare_embedd =  [\n",
        "        f\"{article['title']} {article['description']} {article['content'][0:700]}\"\n",
        "        for article in articles\n",
        "    ]\n",
        "\n",
        "  print(f\"Length of articles to embed: {len(articles_prepare_embedd)}\")\n",
        "  article_embeddings = get_embeddings(articles_prepare_embedd)\n",
        "  return article_embeddings"
      ],
      "metadata": {
        "id": "cNs44lx501Ls"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "similarity_score_func=lambda x, y: 1 - spatial.distance.cosine(x, y)\n",
        "\n",
        "def calculate_cosine_distance(embedding_hypoth, article_embeddings):\n",
        "\n",
        "    cosine_similarities = []\n",
        "    for article_embedding in article_embeddings:\n",
        "        cosine_similarities.append(similarity_score_func(embedding_hypoth, article_embedding))\n",
        "    return cosine_similarities"
      ],
      "metadata": {
        "id": "II7JHwgc2FEJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def sort_articles_by_cosine_similarity(articles, cosine_similarities):\n",
        "    scored_articles = zip(articles, cosine_similarities)\n",
        "    sorted_articles = sorted(scored_articles, key=lambda x: x[1], reverse=True)\n",
        "    print(f\"Top 5 articles scores: {[score for _,score in sorted_articles[0:5]]}\\n\")\n",
        "    # for article, score in sorted_articles[0:5]:\n",
        "    #     print(\"Title:\", article[\"title\"])\n",
        "    #     # print(\"Url:\", article[\"url\"])\n",
        "    #     # print(\"Date of publication:\", article[\"publishedAt\"])\n",
        "    #     # print(\"Description:\", article[\"description\"])\n",
        "    #     # print(\"Content:\", article[\"content\"][0:50] + \"...\")\n",
        "    #     print(\"Score:\", score)\n",
        "    #     print()\n",
        "    return sorted_articles"
      ],
      "metadata": {
        "id": "1aa11gCV3cr7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def context_retrieval(sorted_articles):\n",
        "    \"\"\"Get top 5 articles based on their similarity scores.\"\"\"\n",
        "    formatted_top_results = [article[\"title\"]+\"\\n\"+article[\"description\"]+\"\\n\"+article[\"content\"] for article, _score in sorted_articles[0:5]]\n",
        "\n",
        "    return formatted_top_results\n",
        "\n",
        "\n",
        "def get_final_answer(user_query, formatted_top_results, llm):\n",
        "  \"\"\"Answer the user's question based on the retrieved context using a GPT model: gpt-4o, gpt-4o-mini, gpt-3.5-turbo.\"\"\"\n",
        "  final_input = f\"\"\"\n",
        "  Generate an answer to the user's question based on the given search results.\n",
        "  TOP_RESULTS: {formatted_top_results}\n",
        "  USER_QUESTION: {user_query}\n",
        "\n",
        "  Include as much information as possible in the answer. Reference the relevant search result urls as markdown links.\n",
        "  \"\"\"\n",
        "\n",
        "  completion = client.chat.completions.create(\n",
        "          model=llm,\n",
        "          messages=[\n",
        "              {\"role\": \"user\", \"content\": final_input},\n",
        "          ],\n",
        "      )\n",
        "\n",
        "  return completion.choices[0].message.content\n"
      ],
      "metadata": {
        "id": "e4aCEXg76BF3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Example 1 LLM:"
      ],
      "metadata": {
        "id": "Vlu3M_Vt9Xb9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Similarities against the Hypothetical Answer"
      ],
      "metadata": {
        "id": "00NzHPet5Q-9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "llm = llms[0]\n",
        "embedding_hypoth = get_embeddings(hypoth_answer_llms[llm])[0]\n",
        "article_embeddings = get_embeddings_articles(articles[llm]) #{list of embedded articles , there are 26 articles}\n",
        "cosine_similarities_hypoth = calculate_cosine_distance(embedding_hypoth, article_embeddings)\n",
        "\n",
        "print(f\" len embedding vector={len(embedding_hypoth)}, len artciles embedded={len(article_embeddings)}, len cosine_distance vector={len(cosine_similarities_hypoth)}\")\n",
        "print(cosine_similarities_hypoth[:5])\n",
        "print(\"\\n\")\n",
        "\n",
        "sorted_articles_hypoth = sort_articles_by_cosine_similarity(articles[llm], cosine_similarities_hypoth)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7Lis1URAsf1h",
        "outputId": "16c79efd-45bb-4643-dd43-a1d8b0853fc2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " len embedding vector=1536, len artciles embedded=26, len cosine_distance vector=26\n",
            "[0.8830353301233373, 0.880280016893984, 0.9041821516135816, 0.9036387715042302, 0.8692776540391578]\n",
            "\n",
            "\n",
            "Top 5 articles scores: [0.9057614716323923, 0.905359099913487, 0.9041821516135816, 0.9036387715042302, 0.9022047694353023]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Similarities against the original query"
      ],
      "metadata": {
        "id": "7bTzw0OS9br7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "llm = llms[0]\n",
        "embedding_original_query = get_embeddings(user_query)[0]\n",
        "# article_embeddings = get_embeddings_articles(articles[llm]) #already embedded in the cell before\n",
        "cosine_similarities_original= calculate_cosine_distance(embedding_original_query, article_embeddings)\n",
        "\n",
        "print(f\" len embedding vector={len(embedding_original_query)}, len artciles embedded={len(article_embeddings)}, len cosine_distance vector={len(cosine_similarities_original)}\")\n",
        "print(cosine_similarities_original[:5])\n",
        "print(\"\\n\")\n",
        "\n",
        "sorted_articles_original = sort_articles_by_cosine_similarity(articles[llm], cosine_similarities_original)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ROlTBfXE5pwe",
        "outputId": "b195988b-f9ef-4794-9d46-401cabde4270"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " len embedding vector=1536, len artciles embedded=26, len cosine_distance vector=26\n",
            "[0.8586576952397326, 0.8529322372028058, 0.8813125051656486, 0.8697606885503189, 0.8401190639278262]\n",
            "\n",
            "\n",
            "Top 5 articles scores: [0.8844741226876582, 0.8813125051656486, 0.8789248038009736, 0.8744655908578396, 0.8744095819084349]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Final Answer: Calling LLM to answer the user query"
      ],
      "metadata": {
        "id": "GwA7lXS-vNxZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Against the hypothetical answer and the original user query"
      ],
      "metadata": {
        "id": "oj_v-0wK7sv1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Using the retrieved context coming from the hypothetical answer\n",
        "formatted_top_results_hypoth = context_retrieval(sorted_articles_hypoth)\n",
        "final_answer_hypoth = get_final_answer(user_query, formatted_top_results_hypoth, llm)\n",
        "print(\"Final answer against the Hypothetical query\")\n",
        "display.display(display.Markdown(final_answer_hypoth))\n",
        "\n",
        "#Using the retrieved context coming from the original answer\n",
        "formatted_top_results_original = context_retrieval(sorted_articles_original)\n",
        "final_answer_original = get_final_answer(user_query, formatted_top_results_original, llm)\n",
        "print(\"Final answer against the original query\")\n",
        "display.display(display.Markdown(final_answer_original))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 503
        },
        "id": "SQxNzWaz6Xmd",
        "outputId": "a6dbefa8-5330-409f-b955-85f3e84f8531"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Final answer against the Hypothetical query\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "The global outage caused by CrowdStrike's botched software update had a significant impact on various sectors globally, leading to chaos and disruptions. However, in terms of Microsoft's stock price, the direct impact of CrowdStrike's outage on Microsoft's stock price is not explicitly mentioned in the provided search results.\n\nCrowdStrike's stock fell by 11% due to the software update issue, but there is no specific information on how this influenced Microsoft's stock price. The outage affected users of Microsoft's Windows operating system, which contributed to the widespread disruptions.\n\nFor more details on the impact of CrowdStrike's outage on Microsoft's stock price, you may refer to the search results:\n1. [CrowdStrike CEO Updates Solutions to Global Microsoft Outage](url)\n2. [Microsoft's CrowdStrike leaves business black and blue in India](url)\n\nThese articles may provide further insights into how the global outage caused by CrowdStrike's software update impacted Microsoft and potentially its stock price."
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Final answer against the original query\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "The global outage of CrowdStrike, a cybersecurity company used extensively by Microsoft, had a significant impact on both CrowdStrike's stock price and global systems. CrowdStrike experienced a botched software update that caused chaos around the world, leading to global outages and a drop in its stock price by 11%. This incident affected services for their 30,000 subscribers and prompted a rethink among investors and customers.\n\nAs for Microsoft, the outage caused disruptions in various sectors, including emergency services, medical practices, airlines, banks, and more. The outage was attributed to a dodgy channel file related to CrowdStrike's software update. This had a cascading effect on systems that rely on Microsoft's Windows operating system, leading to further disruptions.\n\nOverall, the outage had a negative impact on CrowdStrike's stock price and operations, as highlighted in the following articles:\n1. [CrowdStrike chaos could prompt rethink among investors, customers](url)\n2. [CrowdStrike shares sink as global IT outage savages systems worldwide](url)\n3. [Microsoft's CrowdStrike leaves business black and blue in India](url)\n\nIt's important to note that the impact on Microsoft's stock price specifically was not explicitly mentioned in the search results provided."
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# All together: With the 3 LLMs"
      ],
      "metadata": {
        "id": "9dKxtKHx7zGM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "llms"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BWKj4HjdaDEz",
        "outputId": "857c0c2d-ab23-41b0-de29-7e8b2c998c5f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['gpt-3.5-turbo', 'gpt-4o-mini', 'gpt-4o']"
            ]
          },
          "metadata": {},
          "execution_count": 209
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "embedding_original_query = get_embeddings(user_query)[0]\n",
        "\n",
        "for llm in llms:\n",
        "  print(llm)\n",
        "  #Hypothetical answer\n",
        "  embedding_hypoth = get_embeddings(hypoth_answer_llms[llm])[0]\n",
        "  article_embeddings = get_embeddings_articles(articles[llm])\n",
        "  cosine_similarities_hypoth = calculate_cosine_distance(embedding_hypoth, article_embeddings)\n",
        "  print(\"Hypothetical Answer: Most relevant News\\n\")\n",
        "  sorted_articles_hypoth = sort_articles_by_cosine_similarity(articles[llm], cosine_similarities_hypoth)\n",
        "  print(\"-\"*50)\n",
        "\n",
        "  #Original Query\n",
        "  cosine_similarities_original= calculate_cosine_distance(embedding_original_query, article_embeddings)\n",
        "  print(\"Original Answer: Most relevant News\\n\")\n",
        "  sorted_articles_original = sort_articles_by_cosine_similarity(articles[llm], cosine_similarities_original)\n",
        "  print(\"-\"*50)\n",
        "\n",
        "  formatted_top_results_hypoth = context_retrieval(sorted_articles_hypoth)\n",
        "  final_answer_hypoth = get_final_answer(user_query, formatted_top_results_hypoth, llm)\n",
        "  print(\"Final answer against the Hypothetical query\")\n",
        "  display.display(display.Markdown(final_answer_hypoth))\n",
        "  print(\"-\"*50)\n",
        "\n",
        "  formatted_top_results_original = context_retrieval(sorted_articles_original)\n",
        "  final_answer_original = get_final_answer(user_query, formatted_top_results_original, llm)\n",
        "  print(\"Final answer against the original query\")\n",
        "  display.display(display.Markdown(final_answer_original))\n",
        "  print(\"-\"*50)\n",
        "  print(\"-\"*50)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "MJXRI6c77yLb",
        "outputId": "47af3d4f-8ab2-4d5b-c402-45b7b8013d59"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "gpt-3.5-turbo\n",
            "Length of articles to embed: 26\n",
            "Hypothetical Answer: Most relevant News\n",
            "\n",
            "Top 5 articles scores: [0.9056245013678469, 0.9054580631940904, 0.9040203721717224, 0.9037450078676516, 0.9022586274192076]\n",
            "\n",
            "--------------------------------------------------\n",
            "Original Answer: Most relevant News\n",
            "\n",
            "Top 5 articles scores: [0.8843513564825746, 0.8812493480390219, 0.8790028253085169, 0.8745070837943586, 0.8743864449395214]\n",
            "\n",
            "--------------------------------------------------\n",
            "Final answer against the Hypothetical query\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "The global outage caused by CrowdStrike's botched software update had a significant impact on various sectors and businesses, including Microsoft. CrowdStrike's software update led to disruptions in services, affecting emergency services, medical practices, airlines, banks, and more worldwide. This resulted in CrowdStrike's stock price plummeting by 19%, prompting concerns among investors and customers.\n\nAs for Microsoft, the outage caused chaos and system snarls, impacting users of Microsoft's Windows operating system. This event had repercussions worldwide, causing disruptions in India with airlines canceling flights, hospitals, banks, and various businesses facing operational challenges. However, the direct impact on Microsoft's stock price is not explicitly mentioned in the search results provided.\n\nFor more detailed information on the impact of the global outage of CrowdStrike on Microsoft's stock price, you can refer to the following search result URLs:\n1. [CrowdStrike chaos could prompt rethink among investors, customers](URL)\n2. [CrowdStrike CEO Updates Solutions to Global Microsoft Outage](URL)\n3. [CrowdStrike shares sink as global IT outage savages systems worldwide](URL)\n4. [Microsoft's CrowdStrike leaves business black and blue in India](URL)"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--------------------------------------------------\n",
            "Final answer against the original query\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "The global outage caused by a botched software update from CrowdStrike had significant repercussions, impacting various sectors worldwide. This incident led to global outages affecting emergency services, medical practices, airlines, banks, and more. CrowdStrike's stock price plummeted by 11% and later by more than 19% amidst the chaos, prompting concerns among investors and customers.\n\nAs for Microsoft, the outage that started by affecting users of Microsoft's Windows operating system was attributed to CrowdStrike's cybersecurity firm. This outage snarled systems from airports to stock exchanges, causing disruptions globally. However, there is no direct mention of the impact on Microsoft's stock price in the provided search results.\n\nFor more information, you can refer to the following links:\n1. [CrowdStrike chaos could prompt rethink among investors, customers](URL)\n2. [CrowdStrike CEO Updates Solutions to Global Microsoft Outage](URL)\n3. [Microsoft's CrowdStrike leaves business black and blue in India](URL)"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "gpt-4o-mini\n",
            "Length of articles to embed: 23\n",
            "Hypothetical Answer: Most relevant News\n",
            "\n",
            "Top 5 articles scores: [0.8985451259633306, 0.89684699586748, 0.8845112396187657, 0.8836237320217197, 0.8829846407367234]\n",
            "\n",
            "--------------------------------------------------\n",
            "Original Answer: Most relevant News\n",
            "\n",
            "Top 5 articles scores: [0.8822393199106383, 0.8679302189191838, 0.867538207728775, 0.8593652185006599, 0.8586576952397326]\n",
            "\n",
            "--------------------------------------------------\n",
            "Final answer against the Hypothetical query\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "The recent global outage resulting from a software update by CrowdStrike, which significantly affected Microsoft systems, led to widespread chaos and operational disruptions across various sectors. However, the specific impact on Microsoft's stock price was not detailed in the search results provided.\n\nThe incident caused millions of Microsoft systems to crash globally, impacting banks, airports, TV stations, grocery stores, and more, highlighting the vulnerabilities in our interconnected digital infrastructure. Reports indicated that many businesses had to switch to cash-only operations or completely shut down until systems were restored. This extensive disruption underscored the risks posed by the reliance on technology and the potential financial ramifications for companies like Microsoft that are intertwined with such systems [source 1](#), [source 2](#), [source 3](#), [source 4](#), [source 5](#).\n\nWhile the search results reflect the broader operational impacts of the outage, including significant consequences for various industries, they did not provide current stock performance metrics or analyses reflecting investor reactions in the wake of the outage. Therefore, for specific information regarding Microsoft's stock price movements or investor sentiment post-outage, one would need to refer to financial news or stock market analysis platforms for real-time updates and insights."
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--------------------------------------------------\n",
            "Final answer against the original query\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "The global outage linked to a faulty update from CrowdStrike, which significantly affected millions of Microsoft systems, caused wide-ranging disruptions across various sectors including banks, airports, and healthcare. However, the search results provided do not specifically address the impact of this incident on Microsoft’s stock price directly.\n\nThe reports detail how the outage led to chaos at airports, grocery stores, and other businesses, and resulted in a choice between going cash-only or closing until systems were restored. Flights were grounded, medical procedures were canceled, and numerous other critical services were impacted by the disruption in IT systems globally. This outage is described as causing significant economic losses and operational paralysis across various industries, highlighting the severity of the situation.\n\nWhile we can infer that such widespread issues could potentially affect investor confidence and, consequently, stock prices, the search results do not provide any specific data or analysis regarding the direct impact on Microsoft’s stock. For a thorough understanding of stock market reactions, one would typically look at financial news sources or stock market analytics which may not have been included in the search results.\n\nFor more details on the outage itself, you can refer to these articles: \n- [The Global CrowdStrike Outage Triggered a Surprise Return to Cash](https://example.com)\n- [Huge Microsoft Outage, Linked to CrowdStrike, Takes Down Computers Around the World](https://example.com)\n- [CrowdStrike IT Outage Cripples the World](https://example.com)\n- [Banks and payments hit as faulty CrowdStrike update causes global Microsoft outage](https://example.com)\n- [Microsoft Outage: CrowdStrike Update Causes Chaos for Flights, Hospitals and Businesses Globally](https://example.com)\n\nIf you're interested in how stock prices reacted, you might want to check financial news outlets or stock tracking services for the latest data and market analysis."
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "gpt-4o\n",
            "Length of articles to embed: 40\n",
            "Hypothetical Answer: Most relevant News\n",
            "\n",
            "Top 5 articles scores: [0.9242878502864379, 0.9098273026927012, 0.9022636887337233, 0.9003755977571286, 0.8961472978984708]\n",
            "\n",
            "--------------------------------------------------\n",
            "Original Answer: Most relevant News\n",
            "\n",
            "Top 5 articles scores: [0.9069811516950537, 0.8862757478801461, 0.885469855492687, 0.8822393199106383, 0.8758189694686986]\n",
            "\n",
            "--------------------------------------------------\n",
            "Final answer against the Hypothetical query\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "Based on the search results, there is no direct mention of the specific impact of the CrowdStrike outage on Microsoft's stock price. However, it's clear that the outage caused by CrowdStrike's software update caused widespread disruptions, affecting airlines, banks, supermarkets, and emergency services ([source](https://www.forbes.com/sites/qaiangel/2023/05/28/crowdstrike-shares-plunge-in-premarket-after-massive-global-it-outage/?sh=30ade44a4ead)).\n\nCrowdStrike's shares fell significantly, plunging by as much as 20% in premarket trading and experiencing a 13% drop as various businesses and services around the globe were disrupted due to the outage linked to their software ([source](https://www.thestreet.com/technology/crowdstrike-shares-fall-more-than-13-as-global-it-outage-grounds-flights-cuts-off-911-access)).\n\nAlthough the specific effect on Microsoft's stock isn't detailed in the search results, the extent of the disruption and the reliance of Microsoft services on CrowdStrike's cybersecurity solutions imply potential repercussions for Microsoft's performance and operational continuity during the outage period. Other sources may need to be consulted to provide a precise impact assessment on Microsoft's stock.\n\nTo explore further details about the involvement of Microsoft and related impact, you may consider examining the following articles: [Crowdstrike shares plunge in premarket after massive global IT outage](https://www.forbes.com/sites/qaiangel/2023/05/28/crowdstrike-shares-plunge-in-premarket-after-massive-global-it-outage/?sh=30ade44a4ead), [CrowdStrike shares fall more than 13% as global IT outage grounds flights, cuts off 911 access](https://www.thestreet.com/technology/crowdstrike-shares-fall-more-than-13-as-global-it-outage-grounds-flights-cuts-off-911-access)."
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--------------------------------------------------\n",
            "Final answer against the original query\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "The global IT outage linked to CrowdStrike, which extensively impacts Microsoft's services, had several significant impacts, but the specific effect on Microsoft's stock price isn't directly mentioned in the provided search results. The outage caused substantial disruptions across various sectors, including airlines, banks, and supermarkets. It led to global chaos, grounding flights, and cutting off 911 access ([source](https://latestinstance.com/news/crowdstrike-shares-fall-more-than-13-as-global-it-outage-grounds-flights-cuts-off-911-access)).\n\nThe chaos also had a ripple effect on the stock market, where the Dow Jones Industrial Average dropped 200 points shortly after the market opened ([source](https://finance.com/news/the-dow-drops-200-points-as-the-crowdstrike-global-tech-outage-hammers-stocks)). This indicates broader market turbulence, possibly affecting many stocks, including Microsoft’s.\n\nHowever, what is reported is the significant plunge in CrowdStrike's shares, falling as much as 20% in premarket trading ([source](https://news.com/crowdstrike-shares-plunge-in-premarket-after-massive-global-it-outage)). Additionally, CrowdStrike's rivals saw a boost in their stock prices as they benefited from the issues CrowdStrike faced ([source](https://marketwatch.com/news/crowdstrike-rivals-getting-stock-boost-from-massive-global-tech-outage)).\n\nGiven these references, while it's clear that the outage affected multiple sectors and caused significant market movements, the exact impact on Microsoft's stock price specifically isn't documented in the provided sources."
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "len(sorted_articles_hypoth)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BBJNRaO4_ug8",
        "outputId": "b4682a29-f291-4e45-f62e-57e35f36dd01"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "26"
            ]
          },
          "metadata": {},
          "execution_count": 112
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Evaluation"
      ],
      "metadata": {
        "id": "K0k9PzgPDspK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will be using DeepEval, to compute 3 metrics:\n",
        "\n",
        "*   Faithfulness\n",
        "*   Context Relevancy\n",
        "*   Anwser Relevancy\n",
        "\n"
      ],
      "metadata": {
        "id": "8wmglABcGerD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install deepeval -q"
      ],
      "metadata": {
        "id": "DI1RKr4R_atT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You need to speficy your OpenAI API key to use DeepEval, in our case.\n",
        "\n",
        "To compute metrics, this library makes several calls to a given LLM, per default they are using GPT-4o.\n",
        "You can use a custom LLM if you want.\n",
        "\n",
        "However note that the under-hood pormpt templates, in the metrics, the LLM is asked to outpout a json format, if you are using a small LLM, this part may not work."
      ],
      "metadata": {
        "id": "c_UUi0NX-lz1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import userdata\n",
        "OPENAI_API_KEY = userdata.get('OPENAI_API_KEY')\n",
        "import os\n",
        "os.environ[ \"OPENAI_API_KEY\" ] = OPENAI_API_KEY"
      ],
      "metadata": {
        "id": "ymxOJWylBI3C"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Faithfullness: Retrieved Context vs LLM's final answer"
      ],
      "metadata": {
        "id": "71SB9xe_DlcR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This evaluates the factual consistency of the **generated answer** relative to the **provided context**.\n",
        "\n",
        "it outputs a **reason** for its **metric score**."
      ],
      "metadata": {
        "id": "49J-NWgautB7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Methodology"
      ],
      "metadata": {
        "id": "Fvq1jDW3uzFc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "1- Use an LLM to break it into statements\n",
        "\n",
        "2- Using an LLM, assert if the statement can or not be inferred from the context ⇒ Verdict: yes or no or idk.\n",
        "\n",
        "3- Compute Faithfulness Score:\n",
        "\n",
        "Faithfulness= Number of Truthful Claims/Total Number of Claims"
      ],
      "metadata": {
        "id": "CKnFtge4uv5Z"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "https://docs.confident-ai.com/docs/metrics-faithfulness"
      ],
      "metadata": {
        "id": "HXkaeoJj_sAt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from deepeval.metrics import FaithfulnessMetric\n",
        "from deepeval.test_case import LLMTestCase"
      ],
      "metadata": {
        "id": "HBMy7CCH_Ys0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def get_faithfulness_metric(user_query,final_answer,formatted_top_results):\n",
        "  metric = FaithfulnessMetric(\n",
        "    threshold=0.7,\n",
        "    model=\"gpt-4o\",\n",
        "    include_reason=True\n",
        "  )\n",
        "  test_case = LLMTestCase(\n",
        "      input= user_query,\n",
        "      actual_output=final_answer,\n",
        "      retrieval_context=formatted_top_results\n",
        "  )\n",
        "\n",
        "  metric.measure(test_case)\n",
        "  score = metric.score\n",
        "  reason = metric.reason\n",
        "  return score, reason"
      ],
      "metadata": {
        "id": "5PnE4gVEFLm6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Original query"
      ],
      "metadata": {
        "id": "Kscu4gznDyfJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "metric = FaithfulnessMetric(\n",
        "    threshold=0.7,\n",
        "    model=\"gpt-4o\",\n",
        "    include_reason=True\n",
        ")\n",
        "test_case = LLMTestCase(\n",
        "    input=user_query,\n",
        "    actual_output=final_answer_original,\n",
        "    retrieval_context=formatted_top_results_original\n",
        ")\n",
        "\n",
        "metric.measure(test_case)\n",
        "print(metric.score)\n",
        "print(metric.reason)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 89,
          "referenced_widgets": [
            "99c810c345144611a574b0354978952d",
            "9beb2254b21c408c905f63b635acef79"
          ]
        },
        "id": "w5gf8aSm_oVj",
        "outputId": "d1aa7af7-ff09-433e-f951-31556ba0b5d0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Output()"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "99c810c345144611a574b0354978952d"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Event loop is already running. Applying nest_asyncio patch to allow async execution...\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Event loop is already running. Applying nest_asyncio patch to allow async execution...\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.8\n",
            "The score is 0.80 because the actual output incorrectly states that CrowdStrike's stock price dropped by 11% when the retrieval context mentions it is down more than 19%, and it also incorrectly claims an outage had a cascading effect on systems relying on Microsoft's Windows operating system, which the retrieval context does not mention.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Hypothetical answer"
      ],
      "metadata": {
        "id": "fbwnfeSAD1SB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import nest_asyncio\n",
        "nest_asyncio.apply()"
      ],
      "metadata": {
        "id": "ZpE_wywQFtta"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "score, reason = get_faithfulness_metric(user_query,final_answer_hypoth,formatted_top_results_hypoth)\n",
        "print(f\"Score: {score}, Reason: {reason}\""
      ],
      "metadata": {
        "id": "kEHn1tL3-P62"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "score, reason = get_faithfulness_metric(user_query,final_answer_original,formatted_top_results_original)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34,
          "referenced_widgets": [
            "e1228e602142478db648aad0ff915991",
            "795101f3f9d242fd9633bbbc8f36f08e"
          ]
        },
        "id": "s7ATKMLqFYFY",
        "outputId": "226adfd0-3ebd-4851-ee69-b3c3cf3cdd26"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Output()"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "e1228e602142478db648aad0ff915991"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Event loop is already running. Applying nest_asyncio patch to allow async execution...\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Event loop is already running. Applying nest_asyncio patch to allow async execution...\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Context Relevancy"
      ],
      "metadata": {
        "id": "uJaOUASBEXmx"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This evaluates how relevant the **retrieved context** is to the **input query**.\n",
        "\n",
        "It outputs a **reason** for its **metric score**."
      ],
      "metadata": {
        "id": "gtgb4qA8EF6a"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Methodology"
      ],
      "metadata": {
        "id": "eUb4FG5XvdmB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "1- Use an LLM to extract statements from the retrieved context\n",
        "\n",
        "2- Using an LLM, assert if each statement is relevant to the input query ==>  yes or no.\n",
        "\n",
        "3- Compute Contextual Relevancy Score:\n",
        "\n",
        "Contextual Relevancy=\n",
        "Number of Relevant Statements/Total Number of Statements​"
      ],
      "metadata": {
        "id": "LHRVxM6fvapB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "https://docs.confident-ai.com/docs/metrics-contextual-relevancy"
      ],
      "metadata": {
        "id": "ZPEg8oo-_uvN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from deepeval.metrics import ContextualRelevancyMetric\n",
        "# from deepeval.test_case import LLMTestCase"
      ],
      "metadata": {
        "id": "pNNNeF8fD4-x"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def get_context_relevancy_metric(user_query,final_answer_hypoth,formatted_top_results_hypoth):\n",
        "  metric = ContextualRelevancyMetric(\n",
        "    threshold=0.7,\n",
        "    model=\"gpt-4o\",\n",
        "    include_reason=True\n",
        "  )\n",
        "  test_case = LLMTestCase(\n",
        "      input= user_query,\n",
        "      actual_output=final_answer_hypoth,\n",
        "      retrieval_context=formatted_top_results_hypoth\n",
        "  )\n",
        "\n",
        "  metric.measure(test_case)\n",
        "  score = metric.score\n",
        "  reason = metric.reason\n",
        "  return score, reason"
      ],
      "metadata": {
        "id": "KBZOXyf1En4J"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Original query"
      ],
      "metadata": {
        "id": "touFdqajX67L"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "score, reason = get_context_relevancy_metric(user_query,final_answer_original,formatted_top_results_original)\n",
        "print(f\"Score: {score}, Reason: {reason}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70,
          "referenced_widgets": [
            "7cbc0bc6cd564bc1b8153b6ec1742cd4",
            "017397fb1d264a55a5aff42cd2600226"
          ]
        },
        "id": "YSkvQICsFDhq",
        "outputId": "7c045b25-3a2a-48d5-8c3e-b1664a237f76"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Output()"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "7cbc0bc6cd564bc1b8153b6ec1742cd4"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Event loop is already running. Applying nest_asyncio patch to allow async execution...\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Event loop is already running. Applying nest_asyncio patch to allow async execution...\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Score: 0.0, Reason: The score is 0.00 because the context focuses on CrowdStrike's stock performance and issues without mentioning Microsoft's stock price.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Hypothetical answer"
      ],
      "metadata": {
        "id": "li6tSnbNX_wL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "score, reason = get_context_relevancy_metric(user_query,final_answer_hypoth,formatted_top_results_hypoth)\n",
        "print(f\"Score: {score}, Reason: {reason}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51,
          "referenced_widgets": [
            "8afc66dbf89641aba9f4041b8b334eee",
            "90b6385e7acb42878af9995a46789a2f"
          ]
        },
        "id": "N23_zB56E_6o",
        "outputId": "3375bf24-d040-46cb-eeed-aa5168146db0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Output()"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "8afc66dbf89641aba9f4041b8b334eee"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Event loop is already running. Applying nest_asyncio patch to allow async execution...\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Event loop is already running. Applying nest_asyncio patch to allow async execution...\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Score: 0.0, Reason: The score is 0.00 because the context discusses CrowdStrike's stock and disruptions but does not mention Microsoft's stock price or its impact.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Answer Relevancy"
      ],
      "metadata": {
        "id": "FU7GNmmcF9g5"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The answer relevancy metric measures the quality of your RAG pipeline's generator by evaluating how relevant the **actual_output** (final answer) of your LLM application is compared to the provided **input**.\n",
        "\n",
        "\n",
        "deepeval's answer relevancy metric is a self-explaining LLM-Eval, meaning it outputs a **reason** for its **metric score**."
      ],
      "metadata": {
        "id": "0UuzUkEsu90S"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "https://docs.confident-ai.com/docs/metrics-answer-relevancy"
      ],
      "metadata": {
        "id": "KxqkMX7y_yi-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from deepeval.metrics import AnswerRelevancyMetric"
      ],
      "metadata": {
        "id": "mvW1NdwBF-Po"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def get_answer_relevancy_metric(user_query,final_answer_hypoth):\n",
        "  metric = AnswerRelevancyMetric(\n",
        "    threshold=0.7,\n",
        "    model=\"gpt-4o\",\n",
        "    include_reason=True\n",
        "  )\n",
        "  test_case = LLMTestCase(\n",
        "      input= user_query,\n",
        "      actual_output=final_answer_hypoth,\n",
        "      # retrieval_context=formatted_top_results_hypoth\n",
        "  )\n",
        "\n",
        "  metric.measure(test_case)\n",
        "  score = metric.score\n",
        "  reason = metric.reason\n",
        "  return score, reason"
      ],
      "metadata": {
        "id": "IfEPq2YAGF0p"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Original query"
      ],
      "metadata": {
        "id": "T_I6JLZ7YHii"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "score, reason = get_answer_relevancy_metric(user_query,final_answer_original)\n",
        "print(f\"Score: {score}, Reason: {reason}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71,
          "referenced_widgets": [
            "9ae13990f4b94589bedfecf87ccfae38",
            "569d02b2e1b14d639d0341fb1cb96e8f"
          ]
        },
        "id": "yR6iyT8aGObp",
        "outputId": "88c71f1d-cd4a-43cd-d1df-aba922404353"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Output()"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "9ae13990f4b94589bedfecf87ccfae38"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Event loop is already running. Applying nest_asyncio patch to allow async execution...\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Event loop is already running. Applying nest_asyncio patch to allow async execution...\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Score: 0.09090909090909091, Reason: The score is 0.09 because the statements focus on CrowdStrike's impact, global systems, and broader disruptions, but fail to specifically address Microsoft's stock price.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Hypothetical answer"
      ],
      "metadata": {
        "id": "2AL18Wy2YKzb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "score, reason = get_answer_relevancy_metric(user_query,final_answer_hypoth)\n",
        "print(f\"Score: {score}, Reason: {reason}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71,
          "referenced_widgets": [
            "b7e41a0b97244238ad271140f9b7755e",
            "029cbff8d3ab423aaf51d2a62f285bf1"
          ]
        },
        "id": "sJ-RgJYJYB5S",
        "outputId": "98abac39-d124-4ee8-b87d-fc1bac5a7bf1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Output()"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "b7e41a0b97244238ad271140f9b7755e"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Event loop is already running. Applying nest_asyncio patch to allow async execution...\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Event loop is already running. Applying nest_asyncio patch to allow async execution...\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Score: 0.8333333333333334, Reason: The score is 0.83 because the output correctly mentions the CrowdStrike outage but includes irrelevant details about CrowdStrike's share price and the disruptions caused by the outage, rather than directly addressing the impact on Microsoft's stock price.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# All together: LLMs + Evaluation"
      ],
      "metadata": {
        "id": "91Otg8-XG9CJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Run All"
      ],
      "metadata": {
        "id": "8O7NUFOaR02W"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def get_all_eval_metrics(user_query,final_answer,formatted_top_results):\n",
        "  score_faithfulness, reason_faithfulness = get_faithfulness_metric(user_query,final_answer,formatted_top_results)\n",
        "  score_cxt_relev, reason_cxt_relev = get_context_relevancy_metric(user_query,final_answer,formatted_top_results)\n",
        "  score_answ_relev, reason_answ_relev = get_answer_relevancy_metric(user_query,final_answer)\n",
        "  print(\"\\nFaithfulness\\n\")\n",
        "  print(score_faithfulness, reason_faithfulness)\n",
        "  print(\"\\nContext Relevancy\\n\")\n",
        "  print(score_cxt_relev, reason_cxt_relev)\n",
        "  print(\"\\nAnswer Relevancy\\n\")\n",
        "  print(score_answ_relev, reason_answ_relev)\n",
        "\n",
        "  return score_faithfulness, reason_faithfulness, score_cxt_relev, reason_cxt_relev, score_answ_relev, reason_answ_relev\n",
        "    # return (score_faithfulness, reason_faithfulness), (score_cxt_relev, reason_cxt_relev), (score_answ_relev, reason_answ_relev)"
      ],
      "metadata": {
        "id": "QjFbAV9XHPdA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# llm = llms[0]\n",
        "\n",
        "embedding_original_query = get_embeddings(user_query)[0]\n",
        "scores_hypoth = {}\n",
        "scores_original = {}\n",
        "\n",
        "for llm in llms:\n",
        "  print(llm)\n",
        "  #Hypothetical answer\n",
        "  embedding_hypoth = get_embeddings(hypoth_answer_llms[llm])[0]\n",
        "  article_embeddings = get_embeddings_articles(articles[llm]) #{list of embedded articles , there are 26 articles}\n",
        "  cosine_similarities_hypoth = calculate_cosine_distance(embedding_hypoth, article_embeddings)\n",
        "  print(\"Hypothetical Answer: Most relevant News\\n\")\n",
        "  sorted_articles_hypoth = sort_articles_by_cosine_similarity(articles[llm], cosine_similarities_hypoth)\n",
        "  print(\"-\"*50)\n",
        "\n",
        "  #Original Query\n",
        "  cosine_similarities_original= calculate_cosine_distance(embedding_original_query, article_embeddings)\n",
        "  print(\"Original Answer: Most relevant News\\n\")\n",
        "  sorted_articles_original = sort_articles_by_cosine_similarity(articles[llm], cosine_similarities_original)\n",
        "  print(\"-\"*50)\n",
        "\n",
        "  formatted_top_results_hypoth = context_retrieval(sorted_articles_hypoth)\n",
        "  final_answer_hypoth = get_final_answer(user_query, formatted_top_results_hypoth, llm)\n",
        "  print(\"#Final answer against the Hypothetical query\")\n",
        "  display.display(display.Markdown(final_answer_hypoth))\n",
        "  print(\"-\"*50)\n",
        "\n",
        "  #Get Evaluations Metrics\n",
        "  scores_hypoth[llm] = get_all_eval_metrics(user_query,final_answer_hypoth,formatted_top_results_hypoth)\n",
        "\n",
        "  formatted_top_results_original = context_retrieval(sorted_articles_original)\n",
        "  final_answer_original = get_final_answer(user_query, formatted_top_results_original, llm)\n",
        "  print(\"#Final answer against the original query\")\n",
        "  display.display(display.Markdown(final_answer_original))\n",
        "\n",
        "  #Get Evaluations Metrics\n",
        "  scores_original[llm] = get_all_eval_metrics(user_query,final_answer_original,formatted_top_results_original)"
      ],
      "metadata": {
        "id": "9xtCwoYJG8Tw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Final Results"
      ],
      "metadata": {
        "id": "Pp1K7q0aRyX-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "scores_original"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_Hb00EJaLJAH",
        "outputId": "4d5f3c45-9a1e-45d9-b406-618bf8c80e2d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'gpt-3.5-turbo': (0.8888888888888888,\n",
              "  'The score is 0.89 because the actual output claims the botched software update affected services for its 30,000 subscribers, which is not mentioned in the retrieval context.',\n",
              "  0.0,\n",
              "  \"The score is 0.00 because the context does not provide any information about Microsoft's stock price, focusing only on the impact of CrowdStrike's outage on other services and its own stock.\",\n",
              "  0.16666666666666666,\n",
              "  \"The score is 0.17 because the output extensively discusses various aspects of CrowdStrike's outage but fails to address the specific impact on Microsoft's stock price.\"),\n",
              " 'gpt-4o-mini': (1.0,\n",
              "  'Fantastic job! The faithfulness score is 1.00 because there are no contradictions between the actual output and the retrieval context. Keep up the great work!',\n",
              "  0.0,\n",
              "  \"The score is 0.00 because the context discusses operational impacts of the CrowdStrike outage but does not provide any information on Microsoft's stock price.\",\n",
              "  0.75,\n",
              "  \"The score is 0.75 because the answer provides relevant information but includes specific examples of industry impact and references for further reading that do not directly address the question about Microsoft's stock price.\"),\n",
              " 'gpt-4o': (1.0,\n",
              "  'The score is 1.00 because there are no contradictions. Great job on maintaining perfect alignment with the retrieval context!',\n",
              "  0.0,\n",
              "  \"The score is 0.00 because the context does not provide any information about the impact of the global outage on Microsoft's stock price.\",\n",
              "  0.6,\n",
              "  \"The score is 0.60 because while the actual output does mention related information about CrowdStrike, it includes several irrelevant details about CrowdStrike and other entities, which do not directly address the impact on Microsoft's stock price.\")}"
            ]
          },
          "metadata": {},
          "execution_count": 160
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "scores_hypoth"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fAixna3aXW5R",
        "outputId": "461555e1-861c-4a76-a81a-d6b08882fb3e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'gpt-3.5-turbo': (0.8,\n",
              "  \"The score is 0.80 because the actual output incorrectly states that CrowdStrike's stock price fell by more than 11%, while the correct figure from the retrieval context is more than 19%.\",\n",
              "  0.0,\n",
              "  \"The score is 0.00 because the context focuses on CrowdStrike's issues and stock performance but does not provide any information about the impact on Microsoft's stock price.\",\n",
              "  0.3,\n",
              "  \"The score is 0.30 because the output contains multiple statements about CrowdStrike and other unrelated information instead of specifically discussing the impact on Microsoft's stock price.\"),\n",
              " 'gpt-4o-mini': (1.0,\n",
              "  'The score is 1.00 because there are no contradictions, indicating that the actual output is perfectly aligned with the retrieval context. Great job!',\n",
              "  0.0,\n",
              "  \"The score is 0.00 because the context does not provide any information about the impact on Microsoft's stock price.\",\n",
              "  0.8571428571428571,\n",
              "  \"The score is 0.86 because while the response addresses the impact on Microsoft's stock price, it includes irrelevant information by directing the reader to external articles rather than focusing solely on the stock price impact.\"),\n",
              " 'gpt-4o': (0.9,\n",
              "  \"The score is 0.90 because the claim states that the global disruption involved Microsoft’s Windows platform, but the retrieval context does not mention Microsoft's Windows platform being involved.\",\n",
              "  0.0,\n",
              "  \"The score is 0.00 because the context only discusses CrowdStrike's stock and the general disruption from the outage, without mentioning Microsoft's stock price or its impact.\",\n",
              "  0.5,\n",
              "  \"The score is 0.50 because while there is some mention of disruptions and impacts across various industries, most of the statements do not directly address the impact on Microsoft's stock price, thus reducing the relevancy.\")}"
            ]
          },
          "metadata": {},
          "execution_count": 207
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import numpy as np"
      ],
      "metadata": {
        "id": "6iWT2KwvLcSv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Against Original Query"
      ],
      "metadata": {
        "id": "sq05prDTXtA7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "scores_original_values = {}\n",
        "for llm in llms:\n",
        "  scores = [score for score in scores_original[llm] if type(score)!= str]\n",
        "  scores_original_values[llm] = scores\n",
        "\n",
        "pd.DataFrame(scores_original_values, index=['faithfulness','context_relevancy','answer_relevancy'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "id": "1lo6yWbILYGH",
        "outputId": "3fa51002-07c0-403b-f504-f93b90f0eb08"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                   gpt-3.5-turbo  gpt-4o-mini  gpt-4o\n",
              "faithfulness            0.888889         1.00     1.0\n",
              "context_relevancy       0.000000         0.00     0.0\n",
              "answer_relevancy        0.166667         0.75     0.6"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-d8cd4e7e-a9d0-4bb7-95b3-40bbd42e8af0\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>gpt-3.5-turbo</th>\n",
              "      <th>gpt-4o-mini</th>\n",
              "      <th>gpt-4o</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>faithfulness</th>\n",
              "      <td>0.888889</td>\n",
              "      <td>1.00</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>context_relevancy</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>answer_relevancy</th>\n",
              "      <td>0.166667</td>\n",
              "      <td>0.75</td>\n",
              "      <td>0.6</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d8cd4e7e-a9d0-4bb7-95b3-40bbd42e8af0')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-d8cd4e7e-a9d0-4bb7-95b3-40bbd42e8af0 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-d8cd4e7e-a9d0-4bb7-95b3-40bbd42e8af0');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-c77dbaef-7ea9-46ac-bd33-4e8d87095642\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-c77dbaef-7ea9-46ac-bd33-4e8d87095642')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-c77dbaef-7ea9-46ac-bd33-4e8d87095642 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"pd\",\n  \"rows\": 3,\n  \"fields\": [\n    {\n      \"column\": \"gpt-3.5-turbo\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.47249447489529905,\n        \"min\": 0.0,\n        \"max\": 0.8888888888888888,\n        \"num_unique_values\": 3,\n        \"samples\": [\n          0.8888888888888888,\n          0.0,\n          0.16666666666666666\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"gpt-4o-mini\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.5204164998665333,\n        \"min\": 0.0,\n        \"max\": 1.0,\n        \"num_unique_values\": 3,\n        \"samples\": [\n          1.0,\n          0.0,\n          0.75\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"gpt-4o\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.5033222956847166,\n        \"min\": 0.0,\n        \"max\": 1.0,\n        \"num_unique_values\": 3,\n        \"samples\": [\n          1.0,\n          0.0,\n          0.6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 204
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "scores_original_raisons = {}\n",
        "for llm in llms:\n",
        "  raisons = [score for score in scores_original[llm] if type(score)== str]\n",
        "  scores_original_raisons[llm] = raisons\n",
        "\n",
        "pd.DataFrame(scores_original_raisons, index=['faithfulness','context_relevancy','answer_relevancy'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "id": "JcGeWIAbgO8I",
        "outputId": "230d8a78-4b90-492e-9c2a-76a1be551171"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                       gpt-3.5-turbo  \\\n",
              "faithfulness       The score is 0.89 because the actual output cl...   \n",
              "context_relevancy  The score is 0.00 because the context does not...   \n",
              "answer_relevancy   The score is 0.17 because the output extensive...   \n",
              "\n",
              "                                                         gpt-4o-mini  \\\n",
              "faithfulness       Fantastic job! The faithfulness score is 1.00 ...   \n",
              "context_relevancy  The score is 0.00 because the context discusse...   \n",
              "answer_relevancy   The score is 0.75 because the answer provides ...   \n",
              "\n",
              "                                                              gpt-4o  \n",
              "faithfulness       The score is 1.00 because there are no contrad...  \n",
              "context_relevancy  The score is 0.00 because the context does not...  \n",
              "answer_relevancy   The score is 0.60 because while the actual out...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-590da71e-73fe-4ac7-8dbe-f04b0d589348\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>gpt-3.5-turbo</th>\n",
              "      <th>gpt-4o-mini</th>\n",
              "      <th>gpt-4o</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>faithfulness</th>\n",
              "      <td>The score is 0.89 because the actual output cl...</td>\n",
              "      <td>Fantastic job! The faithfulness score is 1.00 ...</td>\n",
              "      <td>The score is 1.00 because there are no contrad...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>context_relevancy</th>\n",
              "      <td>The score is 0.00 because the context does not...</td>\n",
              "      <td>The score is 0.00 because the context discusse...</td>\n",
              "      <td>The score is 0.00 because the context does not...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>answer_relevancy</th>\n",
              "      <td>The score is 0.17 because the output extensive...</td>\n",
              "      <td>The score is 0.75 because the answer provides ...</td>\n",
              "      <td>The score is 0.60 because while the actual out...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-590da71e-73fe-4ac7-8dbe-f04b0d589348')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-590da71e-73fe-4ac7-8dbe-f04b0d589348 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-590da71e-73fe-4ac7-8dbe-f04b0d589348');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-2180ad13-97b9-4db8-87a3-a11c1a7c5f45\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-2180ad13-97b9-4db8-87a3-a11c1a7c5f45')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-2180ad13-97b9-4db8-87a3-a11c1a7c5f45 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"pd\",\n  \"rows\": 3,\n  \"fields\": [\n    {\n      \"column\": \"gpt-3.5-turbo\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"The score is 0.89 because the actual output claims the botched software update affected services for its 30,000 subscribers, which is not mentioned in the retrieval context.\",\n          \"The score is 0.00 because the context does not provide any information about Microsoft's stock price, focusing only on the impact of CrowdStrike's outage on other services and its own stock.\",\n          \"The score is 0.17 because the output extensively discusses various aspects of CrowdStrike's outage but fails to address the specific impact on Microsoft's stock price.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"gpt-4o-mini\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"Fantastic job! The faithfulness score is 1.00 because there are no contradictions between the actual output and the retrieval context. Keep up the great work!\",\n          \"The score is 0.00 because the context discusses operational impacts of the CrowdStrike outage but does not provide any information on Microsoft's stock price.\",\n          \"The score is 0.75 because the answer provides relevant information but includes specific examples of industry impact and references for further reading that do not directly address the question about Microsoft's stock price.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"gpt-4o\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"The score is 1.00 because there are no contradictions. Great job on maintaining perfect alignment with the retrieval context!\",\n          \"The score is 0.00 because the context does not provide any information about the impact of the global outage on Microsoft's stock price.\",\n          \"The score is 0.60 because while the actual output does mention related information about CrowdStrike, it includes several irrelevant details about CrowdStrike and other entities, which do not directly address the impact on Microsoft's stock price.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 211
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "index_metrics=['faithfulness','context_relevancy','answer_relevancy']\n",
        "for llm in llms:\n",
        "  print(f\"#{llm}:\")\n",
        "  for i in range(len(index_metrics)):\n",
        "    print(f\"{index_metrics[i]}\")\n",
        "    print(scores_original_raisons[llm][i])\n",
        "  print(\"-\"*50)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IvAD0krbglsB",
        "outputId": "9ddfa359-bffd-4e50-81c0-87e3613b6f38"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "#gpt-3.5-turbo:\n",
            "faithfulness\n",
            "The score is 0.89 because the actual output claims the botched software update affected services for its 30,000 subscribers, which is not mentioned in the retrieval context.\n",
            "context_relevancy\n",
            "The score is 0.00 because the context does not provide any information about Microsoft's stock price, focusing only on the impact of CrowdStrike's outage on other services and its own stock.\n",
            "answer_relevancy\n",
            "The score is 0.17 because the output extensively discusses various aspects of CrowdStrike's outage but fails to address the specific impact on Microsoft's stock price.\n",
            "--------------------------------------------------\n",
            "#gpt-4o-mini:\n",
            "faithfulness\n",
            "Fantastic job! The faithfulness score is 1.00 because there are no contradictions between the actual output and the retrieval context. Keep up the great work!\n",
            "context_relevancy\n",
            "The score is 0.00 because the context discusses operational impacts of the CrowdStrike outage but does not provide any information on Microsoft's stock price.\n",
            "answer_relevancy\n",
            "The score is 0.75 because the answer provides relevant information but includes specific examples of industry impact and references for further reading that do not directly address the question about Microsoft's stock price.\n",
            "--------------------------------------------------\n",
            "#gpt-4o:\n",
            "faithfulness\n",
            "The score is 1.00 because there are no contradictions. Great job on maintaining perfect alignment with the retrieval context!\n",
            "context_relevancy\n",
            "The score is 0.00 because the context does not provide any information about the impact of the global outage on Microsoft's stock price.\n",
            "answer_relevancy\n",
            "The score is 0.60 because while the actual output does mention related information about CrowdStrike, it includes several irrelevant details about CrowdStrike and other entities, which do not directly address the impact on Microsoft's stock price.\n",
            "--------------------------------------------------\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "for llm in llms:\n",
        "  mean_score = np.mean([score for score in scores_original[llm] if type(score)!= str])\n",
        "  print(f\"{round(mean_score,3)} = Mean score for {llm}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rIKiWNlBU_kC",
        "outputId": "0972d793-cc38-42de-ccc4-8d5d944cc2fd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.352 = Mean score for gpt-3.5-turbo\n",
            "0.583 = Mean score for gpt-4o-mini\n",
            "0.533 = Mean score for gpt-4o\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Key Takeaway 1**: gpt-4o-mini shows the best score among the other LLMs. Its score in answer relevancy was better than the one from gpt-4o."
      ],
      "metadata": {
        "id": "ltSMvndEYWSL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Against Hypothetical Answer"
      ],
      "metadata": {
        "id": "o-xZRoTyYTQy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "scores_hypoth_values = {}\n",
        "for llm in llms:\n",
        "  scores = [score for score in scores_hypoth[llm] if type(score)!= str]\n",
        "  scores_hypoth_values[llm] = scores\n",
        "\n",
        "pd.DataFrame(scores_hypoth_values, index=['faithfulness','context_relevancy','answer_relevancy'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "id": "RxHGR6kRXJTU",
        "outputId": "5b883cc0-e4f7-4cac-81ac-5062e13ba330"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                   gpt-3.5-turbo  gpt-4o-mini  gpt-4o\n",
              "faithfulness                 0.8     1.000000     0.9\n",
              "context_relevancy            0.0     0.000000     0.0\n",
              "answer_relevancy             0.3     0.857143     0.5"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-4223f65f-4643-46f0-82e2-9563a456b17b\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>gpt-3.5-turbo</th>\n",
              "      <th>gpt-4o-mini</th>\n",
              "      <th>gpt-4o</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>faithfulness</th>\n",
              "      <td>0.8</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>context_relevancy</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>answer_relevancy</th>\n",
              "      <td>0.3</td>\n",
              "      <td>0.857143</td>\n",
              "      <td>0.5</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-4223f65f-4643-46f0-82e2-9563a456b17b')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-4223f65f-4643-46f0-82e2-9563a456b17b button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-4223f65f-4643-46f0-82e2-9563a456b17b');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-74ee609a-cba3-47e2-b30e-b99636548e14\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-74ee609a-cba3-47e2-b30e-b99636548e14')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-74ee609a-cba3-47e2-b30e-b99636548e14 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"pd\",\n  \"rows\": 3,\n  \"fields\": [\n    {\n      \"column\": \"gpt-3.5-turbo\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.4041451884327381,\n        \"min\": 0.0,\n        \"max\": 0.8,\n        \"num_unique_values\": 3,\n        \"samples\": [\n          0.8,\n          0.0,\n          0.3\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"gpt-4o-mini\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.5408484138857403,\n        \"min\": 0.0,\n        \"max\": 1.0,\n        \"num_unique_values\": 3,\n        \"samples\": [\n          1.0,\n          0.0,\n          0.8571428571428571\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"gpt-4o\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.4509249752822894,\n        \"min\": 0.0,\n        \"max\": 0.9,\n        \"num_unique_values\": 3,\n        \"samples\": [\n          0.9,\n          0.0,\n          0.5\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 205
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "scores_hypoth_raisons = {}\n",
        "for llm in llms:\n",
        "  raisons = [score for score in scores_hypoth[llm] if type(score)== str]\n",
        "  scores_hypoth_raisons[llm] = raisons\n",
        "\n",
        "pd.DataFrame(scores_hypoth_raisons, index=['faithfulness','context_relevancy','answer_relevancy'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "id": "UJwqwI-PhuaQ",
        "outputId": "fd8bb50b-f497-4597-bc5b-d0c47e6679e2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                       gpt-3.5-turbo  \\\n",
              "faithfulness       The score is 0.80 because the actual output in...   \n",
              "context_relevancy  The score is 0.00 because the context focuses ...   \n",
              "answer_relevancy   The score is 0.30 because the output contains ...   \n",
              "\n",
              "                                                         gpt-4o-mini  \\\n",
              "faithfulness       The score is 1.00 because there are no contrad...   \n",
              "context_relevancy  The score is 0.00 because the context does not...   \n",
              "answer_relevancy   The score is 0.86 because while the response a...   \n",
              "\n",
              "                                                              gpt-4o  \n",
              "faithfulness       The score is 0.90 because the claim states tha...  \n",
              "context_relevancy  The score is 0.00 because the context only dis...  \n",
              "answer_relevancy   The score is 0.50 because while there is some ...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-b7634376-3f2e-4864-acc3-987bb8d6850f\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>gpt-3.5-turbo</th>\n",
              "      <th>gpt-4o-mini</th>\n",
              "      <th>gpt-4o</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>faithfulness</th>\n",
              "      <td>The score is 0.80 because the actual output in...</td>\n",
              "      <td>The score is 1.00 because there are no contrad...</td>\n",
              "      <td>The score is 0.90 because the claim states tha...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>context_relevancy</th>\n",
              "      <td>The score is 0.00 because the context focuses ...</td>\n",
              "      <td>The score is 0.00 because the context does not...</td>\n",
              "      <td>The score is 0.00 because the context only dis...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>answer_relevancy</th>\n",
              "      <td>The score is 0.30 because the output contains ...</td>\n",
              "      <td>The score is 0.86 because while the response a...</td>\n",
              "      <td>The score is 0.50 because while there is some ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b7634376-3f2e-4864-acc3-987bb8d6850f')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-b7634376-3f2e-4864-acc3-987bb8d6850f button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-b7634376-3f2e-4864-acc3-987bb8d6850f');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-24da1d7e-5f9c-477f-9df0-821301937729\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-24da1d7e-5f9c-477f-9df0-821301937729')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-24da1d7e-5f9c-477f-9df0-821301937729 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"pd\",\n  \"rows\": 3,\n  \"fields\": [\n    {\n      \"column\": \"gpt-3.5-turbo\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"The score is 0.80 because the actual output incorrectly states that CrowdStrike's stock price fell by more than 11%, while the correct figure from the retrieval context is more than 19%.\",\n          \"The score is 0.00 because the context focuses on CrowdStrike's issues and stock performance but does not provide any information about the impact on Microsoft's stock price.\",\n          \"The score is 0.30 because the output contains multiple statements about CrowdStrike and other unrelated information instead of specifically discussing the impact on Microsoft's stock price.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"gpt-4o-mini\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"The score is 1.00 because there are no contradictions, indicating that the actual output is perfectly aligned with the retrieval context. Great job!\",\n          \"The score is 0.00 because the context does not provide any information about the impact on Microsoft's stock price.\",\n          \"The score is 0.86 because while the response addresses the impact on Microsoft's stock price, it includes irrelevant information by directing the reader to external articles rather than focusing solely on the stock price impact.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"gpt-4o\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"The score is 0.90 because the claim states that the global disruption involved Microsoft\\u2019s Windows platform, but the retrieval context does not mention Microsoft's Windows platform being involved.\",\n          \"The score is 0.00 because the context only discusses CrowdStrike's stock and the general disruption from the outage, without mentioning Microsoft's stock price or its impact.\",\n          \"The score is 0.50 because while there is some mention of disruptions and impacts across various industries, most of the statements do not directly address the impact on Microsoft's stock price, thus reducing the relevancy.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 222
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "index_metrics=['faithfulness','context_relevancy','answer_relevancy']\n",
        "for llm in llms:\n",
        "  print(f\"#{llm}:\")\n",
        "  for i in range(len(index_metrics)):\n",
        "    print(f\"{index_metrics[i]}\")\n",
        "    print(scores_hypoth_raisons[llm][i])\n",
        "  print(\"-\"*50)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zNX2r_Bsh49X",
        "outputId": "9f4aacf4-fc9e-4135-d5a4-4f231308c0dd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "#gpt-3.5-turbo:\n",
            "faithfulness\n",
            "The score is 0.80 because the actual output incorrectly states that CrowdStrike's stock price fell by more than 11%, while the correct figure from the retrieval context is more than 19%.\n",
            "context_relevancy\n",
            "The score is 0.00 because the context focuses on CrowdStrike's issues and stock performance but does not provide any information about the impact on Microsoft's stock price.\n",
            "answer_relevancy\n",
            "The score is 0.30 because the output contains multiple statements about CrowdStrike and other unrelated information instead of specifically discussing the impact on Microsoft's stock price.\n",
            "--------------------------------------------------\n",
            "#gpt-4o-mini:\n",
            "faithfulness\n",
            "The score is 1.00 because there are no contradictions, indicating that the actual output is perfectly aligned with the retrieval context. Great job!\n",
            "context_relevancy\n",
            "The score is 0.00 because the context does not provide any information about the impact on Microsoft's stock price.\n",
            "answer_relevancy\n",
            "The score is 0.86 because while the response addresses the impact on Microsoft's stock price, it includes irrelevant information by directing the reader to external articles rather than focusing solely on the stock price impact.\n",
            "--------------------------------------------------\n",
            "#gpt-4o:\n",
            "faithfulness\n",
            "The score is 0.90 because the claim states that the global disruption involved Microsoft’s Windows platform, but the retrieval context does not mention Microsoft's Windows platform being involved.\n",
            "context_relevancy\n",
            "The score is 0.00 because the context only discusses CrowdStrike's stock and the general disruption from the outage, without mentioning Microsoft's stock price or its impact.\n",
            "answer_relevancy\n",
            "The score is 0.50 because while there is some mention of disruptions and impacts across various industries, most of the statements do not directly address the impact on Microsoft's stock price, thus reducing the relevancy.\n",
            "--------------------------------------------------\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "for llm in llms:\n",
        "  mean_score = np.mean([score for score in scores_hypoth[llm] if type(score)!= str])\n",
        "  print(f\"{round(mean_score,3)} = Mean score for {llm}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mp3Pa_E3WWIb",
        "outputId": "20782138-955c-458e-ef0c-b9495e94d1a0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.367 = Mean score for gpt-3.5-turbo\n",
            "0.619 = Mean score for gpt-4o-mini\n",
            "0.467 = Mean score for gpt-4o\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Key Takeaway 2**: Again, in the hypothetical answer, GPT-4o-mini shows the best score among the other LLMs. Its score in answer relevancy was significantly better than GPT-4o (0.85 vs. 0.5) and even better than GPT-3.5-turbo (0.3). Furthermore, its score in faithfulness was better than the others."
      ],
      "metadata": {
        "id": "vyDJcF_rYnEa"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Key Takeaway 3** : Another important takeway:\n",
        "\n",
        "The score of the results coming from retrieval based on the hypotethical answer (0.619) is better than the one where retrieval is based on the original query (0.583), when using gpt-4o-mini. This higlights the fact that the re-ranking process leads to better results."
      ],
      "metadata": {
        "id": "l7Y4C_CrZSWi"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Next:**\n",
        "\n",
        "**Retrieval Context:**\n",
        "\n",
        "Even if gpt-4o-mini is showing a good performance, however, the context relevance metric is 0 for all LLMs. This part needs to be reworked again.\n",
        "In the retrieval part, I took title + description and the beginning of the content. That was not enough.\n",
        "A good way needs to be : Parsing the whole html for each article, and gathering all this information together, chunking it in a given size\n",
        "\n",
        "**DeepEval and gpt-4o-mini:**\n",
        "\n",
        "It could be interesting to run evaluation metrics with gpt-4o-mini instead of gpt-40. Because the underhood calculation of the scores in the evaluation metrics are based on templated prompts and the capability of the LLM to well compare a given claim/statement in the retrieved context (for example) vs the final answer.\n",
        "\n",
        "This leads me to this conclusion, because I was not expecting gpt-4o-mini to outperfom gpt-4o!!"
      ],
      "metadata": {
        "id": "zRErXfZvaeYR"
      }
    }
  ]
}